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Text-Dependent  Speaker  Verification  Using 
Vector  Quantization  Source  Coding 


I.  INTRODUCTION 

Speaker  verification  by  machine  consists  of  automatically  authenticating  the  identity 
claimed  by  a  speaker  given  only  samples  of  the  speaker’s  voice.  It  has  been  an  area  of  active 
study  for  more  than  twenty  years,  and  two  categories  of  approaches  to  this  problem  have 
developed.  In  one,  verification  decisions  are  based  on  speech  that  is  selected  by  the  speaker  and 
not  known  ahead  of  time  by  the  verification  system.  This  is  known  as  text-independent 
verification.  In  the  other  category,  the  verification  system  is  trained  on  a  prespecified  utterance 
and  later  this  same  utterance  is  spoken  by  the  individual  in  question  -  this  is  called  text- 
dependent  speaker  verification.  In  this  paper,  we  describe  and  evaluate  several  new  approaches  to 
the  text-dependent  speaker  verification  problem. 

A  typical  approach  to  text-dependent  speaker  verification  consists  of  the  following.  Select  a 
parameter  or  a  set  of  parameters  that  can  be  derived  from  the  speech  waveform  and  then 
represent  each  speaker  by  a  time-series  of  these  parameters  (called  a  reference  template)  obtained 
from  a  particular  utterance.  The  parameters  are  chosen  normally  with  the  hope  that  they  reflect 
speaker-specific,  organic  differences  in  the  structure  of  the  vocal  apparatus  or,  perhaps,  that  the 
time  series  of  parameters  will  reflect  learned  differences  in  the  use  of  the  vocal  apparatus  to  pro¬ 
duce  a  particular  utterance.  After  obtaining  a  reference  for  each  speaker,  an  unknown  speaker 
claims  an  identity  and  speaks  an  appropriate  utterance.  This  utterance  is  analyzed  and  a  time- 
series  of  parameters  is  obtained.  The  unknown  speaker’s  parameters  are  aligned  in  time  with  the 
reference  stored  for  the  speaker  whose  identity  was  claimed,  and  the  decision  to  accept  or  reject 
the  speaker  is  based  on  a  measure  of  the  similarity  between  the  two  time-series  of  parameters. 
Examples  of  parameters  that  have  been  used  in  this  way  are  pitch  [l],  short  time  energy  [l],  short 
time  spectra  [2],  and  linear  predictive  coding  (LPC)  coefficients  or  parameters  that  can  be  derived 
from  these  coefficients  [3] . 

In  addition  to  the  template  matching  approach  described  above,  statistical  methods  have 
been  studied  [4,  5j.  These  methods  use  large  amounts  of  training  data  to  estimate  the  underlying 
probability  densities  for  the  parameters  chosen  to  represent  a  speaker.  Once  the  probability  den¬ 
sities  are  specified,  statistical  detection  theory  methods  are  used  to  verify  a  speaker  [6] . 

We  approach  the  text-dependent  speaker  verification  problem  from  a  different  viewpoint. 
We  consider  a  speaker  of  a  particular  utterance  as  an  information  source,  and  we  model  this  infor¬ 
mation  source  using  a  standard  information-theoretic  source  coding  method  called  vector  quanti¬ 
zation  (VQ).  VQ  is  a  source  coding  technique  [7]  that  has  been  used  successfully  in  both  speech 
coding  (8j  and  speech  recognition  [9,  10,  11].  In  VQ,  each  source  vector  is  coded  as  one  of  a  pre¬ 
stored  set  of  codewords,  called  a  codebook,  by  finding  the  codeword  that  minimizes  the  distortion 
between  itself  and  the  source  vector.  For  speech,  a  codebook  is  designed  from  a  training  sequence 
consisting  of  typical  speech  [12].  The  training  sequence  is  divided  into  frames  (typically  20  mil¬ 
liseconds),  linear  predictive  analysis  is  done  on  each  frame,  and  a  clustering  algorithm  is  used  on 
this  sequence  of  LPC  coefficients  to  obtain  a  codebook  of  representative  spectra,  or  codewords. 
The  codebook  is  designed  to  minimize  the  average  quantization  distortion  between  itself  and  the 
training  sequence. 
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To  use  VQ  source  coding  in  speaker  verification,  we  represent  each  speaker  by  a  VQ  code¬ 
book  designed  from  a  training  sequence  composed  of  repetitions  of  a  particular  utterance.  Later, 
this  same  utterance  is  spoken  by  an  unknown  speaker  with  a  claimed  identity.  This  test  utter¬ 
ance  is  coded  in  the  codebook  representing  the  speaker  whose  identity  was  claimed,  and  the 
resulting  quantization  distortion  is  compared  to  a  threshold.  If  the  distortion  is  less  than  the 
threshold,  the  speaker  is  accepted. 

In  addition  to  our  source  coding  point  of  view,  our  speaker  verification  approach  is  quite 
different  from  other  approaches  in  several  ways.  No  attempt  is  made  to  align-in-time  a  test 
sequence  with  a  stored  reference  sequence  (indeed,  no  reference  sequence  exists),  and  no  explicit 
estimate  is  made  of  an  underlying  probability  density  function.  Our  verification  procedures  are, 
however,  closely  related  to  optimal  information-theoretic  methods  of  classification  that  use  the 
information  dissimilarity  between  two  vectors  as  a  discrimination  measure  [13,  14].  Preliminary 
results  were  reported  in  [15]. 

We  previously  used  VQ  source  coding  in  isolated  word  recognition  [9,  16,  17].  The  methods 
used  in  those  approaches  to  represent  a  word  (design  a  codebook)  and  to  compare  an  unknown 
input  word  with  the  stored  codebooks  (classify  an  input  utterance)  are  the  same  as  the  ones 
described  in  this  paper  to  represent  and  verify  a  speaker.  The  differences  are  in  the  application  of 
the  ideas  and  in  the  use  of  thresholds  to  make  decisions. 

Recently,  another  VQ  based  approach  to  speaker  identification  has  been  reported  by  Soong 
et  al  [18].  In  this  approach,  each  speaker  is  represented  by  a  single  VQ  codebook  designed  to 
represent  that  speaker  saying  the  10  digits.  An  unknown  speaker  then  says  the  10  digits,  and  the 
average  quantization  distortion  resulting  from  encoding  the  digits  is  used  as  a  discrimination 
measure  to  identify  the  speaker.  Reported  results  are  quite  good  -  an  error  rate  less  than  2% 
[18]. 

The  rest  of  this  paper  is  organized  as  follows.  Section  II  describes  three  ways  to  represent  a 
speaker  by  using  VQ  source  coding.  Section  III  explains  our  speaker  verification  approach.  Sec¬ 
tion  IV  presents  experimental  results,  and  section  V  concludes  with  a  summary  and  general  dis¬ 
cussion. 

H.  BACKGROUND 

In  this  section  we  briefly  describe  three  ways  to  design  a  source  model  of  a  speaker;  for 
detailed  descriptions  of  the  methods,  see  [9,  16,  17).  Following  these  descriptions  is  a  list  of  the 
distortion  measures  and  LPC  parameters  we  used  in  the  speaker  verification  experiments. 

First  we  establish  some  notation  and  define  some  terms.  Upper  and  lower  case  roman  and 
italic  letters  (e.g.  n,  N,  q,  Q)  denote  scalars;  lower  case  italic  letters  with  bars  (e.g.  7)  denote 

vectors;  upper  case  italic  letters  with  bars  (e.g.  C)  denote  sets  of  vectors  (e.g. 

C  =  (e-! ,72,  ■  ■  ■  ,7# });  bold  lower  case  roman  letters  (e.g.  c)  denote  sequences  of  vectors  (e.g. 

c  =  e) ;  i=l,  •  •  •  ,K);  and  bold  upper  case  roman  letters  (e.g.  C)  denote  sets  of  vector 

sequences  (e.g.  C  =  (c  1(c  n,  •  •  •  ,c  lV  })• 

Throughout,  all  vectors  consist  of  LPC  coefficients  and  a  gain  term.  T  =  {<1,^2,  '  '  ’  Jp} 
is^  a  P -vector  training  sequence  obtained  from  M  repetitions  of  an  utterance  by  a  speaker. 
V  =  (<7i,u2,  •  •  •  ,7i  }  is  an  L  -vectoi_test  sequence  corresponding  to  an  utterance  obtained  from 
a  speaker  for  verification  purposes.  C  represents  a  VQ  codebook,  whether  it  is  single-section  or 
multisection  will  be  clear  from  the  context,  and  finally,  C  represents  a  matrix  quantization  code¬ 
book. 

A.  Single  Section  Vector  Quantization 

For  speaker  verification,  a  single-section  VQ  codebook  C  is  designed  to  minimize  the  aver¬ 
age  distortion  that  results  from  encoding  a  training  sequence  T 
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where  e$  is  the  codeword  resulting  from  encoding  the  speech  segment  tp  , 
(f"p  .«>  )  =  mini  (Tp  ,T{ ), 
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and  d  is  an  appropriate  vector  distortion  measure.  This  codebook  represents  a  speaker  saying  a 
particular  word. 


The  average  quantization  distortion  Davg  that  results  from  coding  a  verification  utterance  V 
in  codebook  C  is 
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It  is  this  average  quantization  distortion  that  is  used  in  making  the  verification  decision. 

This  approach  is  called  single  section  to  distinguish  it  from  the  approach  described  in  the 
next  section  in  which  each  speaker  is  represented  by  a  codebook  consisting  of  a  sequence  of 
single-section  codebooks. 


B.  Multisection  Vector  Quantization 


In  multisection  VQ,  we  represent  each  speaker  by  a  time-dependent  sequence  of  single  sec¬ 
tion  codebooks,  which  we  call  a  multisection  codebook.  A  speaker  is  verified  by  dividing  his 
verification  utterance  V  into  sections  that  correspond  to  the  sections  of  the  multisection  code¬ 
books,  doing  VQ  on  a  section-by-section  basis  with  the  appropriate  multisection  codebook,  and 
computing  the  average  distortion. 

To  be  more  specific,  let  Fq  be  the  number  of  frames  in  the  qtk  utterance  in  the  training 
sequence  for  G ,  where  q  =1,  •  •  •  ,M;  and  let  U, ^  be  the_m‘*  frame  in  the  qtk  training  utter¬ 
ance  where_w  =1  ,...,Fq  .  Now  the  multisection  codebook  C  consists  of  a  sequence  of  VQ  section 
codebooks  Cj ,  where  the  section  codebook  Cj  is  designed  using  (1)  and  n  frames  from  each  train¬ 
ing  utterance.  That  is,  C}_ is  designed  from  the  frames  Umq  ,  where  m  =(;-l)n  +1 jn  ,  and 

q=L,...,M.  For  example,  C  t  is  designed  from  the  first  n  frames  of  each  training  utterance,  C2 
from  the  second  n  frames,  etc.  We  call  n  the  section  length  -  it  is  the  number  of  frames  that  are 
spanned  per  section.  Finally,  let  <T^ ,  i  =1,  .  .  .  ,  Nj  be  codewords  in  section  codebook  Cj  . 

Darg  Js  the  average  distortion  resulting  from  coding  the  verification  utterance  V  with  the 
codebook  C, 

Davg  =  ~r  Yj  >  (3) 

L  ,  -i 

where  5  is  the  number  of  section  codebooks  in  C , 

rain  ,  L  | 

di  =  E  min  d  (v,  ,Cjj ), 

is  the  total  distortion  from  coding  the  jtk  section  of  the  utterance  V  with  the  jtk  section  code¬ 
book  Cj  of  C,  and  n  is  the  section  length.  The  verification  decision  is  made  using  this  distor¬ 
tion. 

C.  Matrix  Quantization 

In  matrix  quantization,  instead  of  coding  a  single  source  vector  in  a  codebook  containing 
characteristic  vectors,  we  code  a  time-ordered  sequence  of  source  vectors  in  a  codebook  containing 
characteristic  vector  sequences.  Given  T,  we  find  the  matrix  quantization  codebook  C  contain¬ 
ing  codeword  matrices  c  —  [c'.1,c‘.-2,...,e\K-  ]  that  minimizes 
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where  e  ^  is  the  codeword  matrix  resulting  from  coding  the  sequence  of  training  vectors 

=  [^p.^p+ 1,- --ifp 

by  using  the  nearest  neighbor  rule 
£>(t  ,c  fl)  =  min  D( t  ,c  ;), 

3 

and  where  the  distortion  between  a  speech  segment  t  and  the  jtk  codeword  is 

K  __ 

0  (*»«>)  =  E  diti,cJi)-  (4) 

i  -l 

We  call  K  the  codeword  matrix  size.  The  MQ  codebook  design  algorithm  vie  used  [19]  is  a  gen¬ 
eralized  version  of  the  VQ  design  algorithm  developed  by  Linde  et  al  [12]. 

To  use  MQ  in  speaker  verification,  we  represent  a  speaker  saying  a  particular  word  by  a 
codebook  C  ,  just  as  in  the  VQ  approaches  above.  A  verification  utterance  is  processed  by  divid¬ 
ing  it  into  overlapping  sequences  of  K  frames,  coding  each  K  frame  sequence  in  the  speaker- 
codebook  C  ,  and  computing  the  average  quantization  distortion  between  the  utterance  and  the 
codebook.  To  be  specific,  for  a  verification  utterance  V ,  the  average  distortion  resulting  from 
coding  it  with  codebook  C  is 

«  L-K+i 

A.,,  =  T  £  ^  (v  i , e  a ).  (5) 

v  i-i 

D.  Distortion  Measures 

Based  on  results  from  previous  work  on  isolated  word  recognition  [9],  we  used  the  gain  nor¬ 
malized  Itakura-Saito  distortion  measure  (daN)  in  (1)  and  (4)  to  generate  codebooks.  For  power 
spectrum  estimates  /  and  /  that  have  the  autoregressive  (LPC)  form 

1  {9)  =  M(*)|8"' 

where 

A(z)  =  E 

k—0 


and  z=exp(*0),  the  dGN  distortion  is  given  by 


das(f  ,f)  =  - 5r-l, 
<r 


where 

a  =  r  (0)r„  (0)  +  2  r(n)rt{n), 

n  *1 

M-n  *  . 
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and  where  r(n)  are  the  time-domain  autocorrelations  of  /  (0).  For  the  verification  distortion 
measure  in  (2),  (3),  and  (5),  we  used  the  gain  optimized  Itakura-Saito  distortion  measure  ( dGo)> 

*ao{f  ,/)  =  ln(a)-ln(<r), 

which  is  also  known  as  the  log  likelihood  distortion  measure.  Properties  of  these  distortion  meas¬ 
ures  are  discussed  in  [20]. 
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E.  LPC  Parameters 


LPC  parameters  for  both  codebook  generation  and  speaker  verification  were  generated  using 
the  autocorrelation  method  of  linear  predictive  analysis  with  Hamming  windowing.  We  chose 
analysis  conditions  for  compatibility  with  the  Navy’s  2.4-kbs  LPC-10  system[2l]:  analysis  window 
width  =  128  points,  filter  order  =  10,  and  pre-emphasis  =  94%. 

m.  SPEAKER  VERIFICATION  APPROACH 


Usually,  no  information  is  available  for  the  characteristics  of  specific  unacceptable  speakers, 
and  the  main  problem  in  applying  these  source  coding  approaches  to  speaker  verification  is  to  for¬ 
mulate  a  criterion  for  rejecting  a  speaker.  To  decide  whether  to  reject  a  speaker  (given  an  utter¬ 
ance),  we  associate  a  threshold  with  each  speaker  codebook.  An  unknown  utterance  (speaker)  is 
rejected  if  its  distortion  exceeds  the  threshold.  To  design  thresholds  for  a  speaker,  we  estimate 
parameters  for  two  Gaussian  distributions:  the  in-class  distribution  of  distortions  (obtained  by 
encoding  utterances  from  that  speaker  in  his  or  her  codebook)  and  the  out-of-class  distribution  of 
distortions  resulting  from  encoding  utterances  spoken  by  other  speakers.  We  choose  the  threshold 
to  equalize  the  overlap  area  of  the  two  distributions,  thus  equalizing  the  expected  numbers  of 
imposter  acceptances  (false  acceptances)  and  rejections  of  acceptable  speakers  (false  rejections). 

In  more  detail,  the  threshold  computation  is  as  follows.  For  each  speaker,  encode  that 
speaker’s  training  data  with  his  or  her  codebook.  Compute  the  mean  distortion  fi,'a  resulting  from 
encoding  the  training  data  from  speaker  i  in  speaker  « ’s  codebook,  and  compute  the  correspond¬ 
ing  standard  deviation  <r,m.  Also  compute  /itoul,  the  mean  distortion  resulting  from  encoding  utter¬ 
ances  not  spoken  by  speaker  »  using  the  codebook  for  speaker  i ,  and  the  corresponding  standard 
deviation  <r  °u‘.  To  equalize  the  number  of  false  acceptances  and  false  rejections,  the  threshold  T, 
is  chosen  to  be  an  equal  number  of  standard  deviations  away  from  each  mean,  giving 


/i/vr + 

<rrl  +  *,m 


This  method  of  threshold  determination  assumes  Gaussian  distributions.  Some  previous  studies  by 
Buck  [22j,  however,  showed  that  the  logarithms  of  average  distortions  are  more  nearly  Gaussian 
than  the  distortions  themselves;  so  the  thresholds  were  based  on  the  statistics  of  the  logarithms  of 
distortions,  instead  of  simply  the  distortion  as  shown  in  (2),  (3),  and  (5). 

To  verify  a  speaker,  the  verification  utterance  V  is  coded  in  the  appropriate  codebook  and 
the  average  log  distortion  is  computed.  This  distortion  value  is  compared  to  the  threshold  associ¬ 
ated  with  that  codebook,  and  if  the  distortion  value  exceeds  the  threshold,  the  speaker  is  rejected; 
otherwise  the  speaker  is  accepted. 


Preliminary  experiments  indicated  that  verification  accuracy  using  a  single  verification 
utterance  is  poor  [15].  To  improve  the  verification  accuracy,  we  based  the  verification  decision  on 
the  results  for  several  words.  The  next  section  describes  our  approach  to  extending  this  method 
to  multiple  words. 


A.  Extension  To  Multiple  Words 


In  previous  work  [15],  we  examined  three  ways  of  extending  our  method  to  more  than  one 
word.  All  three  methods  achieved  about  the  same  verification  accuracy,  and  based  on  those 
results,  we  used  the  simplest  of  the  three  methods  in  this  work. 

For  each  speaker,  a  separate  codebook  is  designed  for  each  word;  if  W  words  are  to  be  spo¬ 
ken,  there  are  W  codebooks  for  each  speaker.  Separate  thresholds  are  computed  for  each  word, 
and  W  different  verification  decisions  are  made.  For  example,  if  a  speaker  is  requested  to  say 
zero  ,  three  ,  and  nine  ,  the  zero  utterance  is  encoded  with  the  zero  codebook  from  that  speaker; 
the  three  utterance  is  encoded  with  the  three  codebook;  etc.  To  make  a  verification  decision,  we 
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use  a  majority  rule;  the  decision  made  by  a  majority  of  the  individual  word  classifiers  is  used  as 
the  overall  decision.  In  case  of  ties,  the  speaker  is  rejected. 

IV.  EXPERIMENTS 

We  first  describe  the  speech  data  bases  that  were  used  in  the  verification  experiments.  We 
next  describe  how  the  data  bases  were  partitioned  for  use  in  separate  parameter  studies  and 
evaluation  tests  and  what  parameters  were  varied  in  the  studies.  This  is  followed  by  three  sub¬ 
sections;  each  subsection  describes  the  verification  results  using  one  of  the  source  models 
described  in  section  II. 

A.  Data  Bases 

We  combined  two  data  bases  to  do  these  experiments,  both  collected  by  Texas  Instruments 
Inc.  (TI).  The  main  difference  in  the  data  bases  is  the  resolution  of  the  A/D  converters.  One 
data  base  was  digitized  with  a  12-bit  con  verter;  the  second  was  digitized  with  a  16-bit  converter. 

Data  for  designing  the  codebooks  to  represent  the  speakers,  determining  the  parameters  for 
the  in-class  distributions,  and  testing  verification  accuracy  came  from  the  data  base  described  in 
[23].  It  contains  26  utterances  of  each  digit  { zero  through  nine)  by  16  speakers  (8  male  and  8 
female).  We  call  this  data  base  TI-1.  The  data  used  for  determining  the  parameters  for  the  out- 
of-class  distributions  and  for  testing  the  imposter  rejection  capabilities  of  the  methods  came  from 
a  data  base  designed  for  evaluating  speaker-independent  recognition  of  the  digits  [24],  It  contains 
two  utterances  of  each  of  the  10  digits  from  109  adult  male  and  111  adult  female  speakers  that 
are  distinct  from  the  speakers  in  TI-1.  This  data  base  is  divided  into  two  parts:  a  training  part 
containing  54  male  and  55  female  speakers,  and  a  testing  part  containing  54  male  and  57  female 
speakers.  We  call  this  second  data  base  TI-2. 

Automatic  endpoint  detection  for  both  training  and  test  utterances  was  used  in  our  experi¬ 
ments.  Our  endpoint-detection  algorithm  is  based  on  ideas  presented  in  [25,  26].  Briefly,  the 
algorithm  first  analyzes  the  background  noise  to  determine  its  average  magnitude  and  then  uses 
the  result  to  set  various  thresholds  that  are  used  to  find  significant  “energy  clumps”  in  the  data. 
Se*>  [9]  for  more  details. 

B.  Data  Base  Partition 

We  first  determined  the  number  of  training  utterances  required  to  characterize  a  speaker 
saying  a  digit.  To  do  this,  for  each  speaker-digit  combination,  we  designed  a  series  of  8- 
codeword,  single  section  codebooks.  We  designed  the  first  codebook  using  a  one-utterance  train¬ 
ing  sequence,  and  increased  the  number  of  training  utterances  by  one  for  each  new  codebook.  We 
recorded  the  average  codebook-design  distortion  for  each  codebook,  and  after  designing  all  the 
codebooks,  examined  the  results  looking  for  the  number  of  training  utterances  required  to  maxim¬ 
ize  the  codebook-design  distortion.  (See  Figure  1  for  the  results  from  a  typical  speaker.)  On  aver¬ 
age,  it  took  8  utterances  to  reach  90%  of  the  maximum  codebook-design  distortion,  and  based  on 
this,  we  designed  codebooks  using  8  training  utterances  in  all  our  experiments. 

In  all  three  parameter  studies  described  below,  we  designed  digit  codebooks  for  each  speaker 
in  the  TI-1  data  base  from  the  first  8  utterances  of  each  digit.  These  8  training  utterances  plus 
the  next  4  utterances  were  used  to  estimate  the  parameters  for  the  in-class  distribution  for  each 
speaker-word  model.  The  next  7  digits  in  TI-1  were  the  speaker  supplied  verification  data.  For 
the  parameter  studies,  the  TI-2  training  data  was  divided  into  two  parts.  One  part,  containing  the 
first  27  male  and  27  female  speakers,  was  used  to  estimate  the  out-of-class  distribution  parame¬ 
ters;  the  second  part,  containing  the  rest  of  the  training  portion  of  TI-2,  was  used  as  imposter 
data. 

Based  on  the  results  of  the  parameter  studies,  we  chose  several  sets  of  words  and  codebook 
parameters  to  use  in  the  full  data  base  tests  of  the  three  source  models.  In  these  tests,  the 


training  data  for  each  speaker  codebook  again  consisted  of  the  first  8  utterances  of  a  digit.  The 
m-class  parameter  estimation  data,  however,  consisted  of  the  first  16  utterances.  The  remaining 
10  utterances  of  each  digit  were  the  verification  data.  We  used  all  109  speakers  in  the  training 
portion  of  TI-2  to  estimate  the  out-of-class  distribution  parameters  and  the  111  speakers  in  the 
test  portion  as  the  imposters. 

C.  Experimental  Parameters 

The  codebook  size,  or  the  number  of  codewords  in  a  codebook,  is  a  parameter  that  we 
varied  in  the  experiments.  For  single  section  codebooks,  the  codebook  size  is  always  a  power  of  2 
-  i.e.,  N  =  2fi  ,  and  we  call  R  the  rate  of  the  codebook.  For  multisection  codebooks,  the  size  of 
the  constituent  section  codebooks  is  also  always  a  power  of  2,  and  we  call  the  section  codebook 
rate  Rs .  Similarly,  the  size  of  matrix  quantization  codebooks  is  a  power  of  2;  we  call  the  matrix 
codebook  rate  RM . 

In  addition  to  the  codebook  rates,  we  varied  the  section  length  n  for  multisection  codebooks 
and  the  matrix  size  K  for  matrix  quantization  codebooks  during  the  parameter  studies.  The 
parameters  used  during  verification  always  matched  those  used  in  designing  the  codebooks. 

There  are  a  number  of  factors  affecting  the  design  of  codebooks  and  thus  the  verification 
results  that  we  did  not  vary.  For  one,  we  preprocessed  the  training  and  verification  data  by  divid¬ 
ing  each  utterance  into  24  equal  length  frames.  This  was  done  to  provide  a  rough  form  of  nor¬ 
malization.  Also,  for  single-section  and  multisection  codebooks,  we  used  an  energy  (sum-of- 
squares  of  data  points)  threshold  of  250  to  ignore  low  energy  frames;  this  threshold  was  used  both 
in  codebook  generation  and  speaker  verification.  For  matrix  quantization,  we  bandied  low  energy 
frames  in  the  following  manner.  The  first  K  -1  low-energy  frames  in  a  sequence  were  replaced 
with  flat-spectrum  frames  with  energy  equal  to  250;  if  more  than  K  -1  frames  occurred  in  a 
sequence,  we  ignored  all  but  the  first  K- 1.  The  reason  for  this  was  to  preserve  transitions  from 
silence-to-speech  and  vice  versa,  while  eliminating  any  all-silent  training  and  verification  seg¬ 
ments. 

D.  Single  Section  Results 

Parameter  Studies.  We  varied  the  codebook  rate  R  in  these  experiments.  Verification  deci¬ 
sions  were  made  using  all  10  digits  and  the  majority-rule  classifier  that  was  described  in  section 
III.A.  The  results  are  listed  in  Table  I  for  R  ranging  from  1  to  4.  Most  of  the  verification  errors 
were  false  rejections.  This  implies  that  the  acceptance  thresholds  are  too  small.  Because  of  this, 
in  the  verification  tests  (described  in  this  and  the  next  two  subsections  IV.E  and  IV. F),  we 
increased  the  number  of  training  utterances  used  to  estimate  the  in-class  distribution  parameters. 
For  R  equal  to  3  and  4,  all  errors  were  caused  by  just  2  speakers. 

For  R  equal  to  3,  we  measured  the  verification  accuracy  of  each  digit  individually;  the 
results  are  in  Table  II.  No  single  digit  reliably  verifies  the  speakers,  and  the  individual  digit 
results  are  also  biased  toward  false  rejections.  The  last  column  in  Table  II  contains  the  square 
root  of  the  product  of  the  false-acceptance  rate  and  the  false-rejection  rate  (\/ FA  *FR  );  this  is 
considered  a  good  overall  performance  measure  [27]. 

Verification  Tests.  As  noted  above,  we  added  more  utterances  to  the  data  that  was  used  to  esti¬ 
mate  the  in-class  distribution  parameters;  the  new  training  set  contained  16  utterances.  We  felt 
that  by  including  more  utterances  that  were  not  in  the  codebook  training  set,  the  in-class  distri¬ 
bution  for  a  speaker  would  better  represent  new  utterances  from  that  speaker.  Rate-3  codebooks 
were  used  in  the  verification  tests  because  they  did  best  in  the  parameter  study  and  also  because 
rate-3  codebooks  yielded  good  speaker-trained  isolated  word  recognition  results  [9]. 

The  results,  using  all  10  digits  in  the  verification  decision,  are  listed  by  individual  speaker  in 
Table  III.  The  majority  of  the  errors  were  caused  by  KAB  and  GRD;  these  also  were  the  two 
difficult  speakers  in  the  parameter  study.  The  results  are  still  biased  toward  false  rejections, 


although  the  bias  is  smaller  than  it  was  in  the  parameter  study.  VFA  *FR  for  this  test  was  0.9. 

Next  several  subsets  of  the  digits  were  tested,  each  consisting  of  5  digits.  Based  on  the  sin¬ 
gle  digit  parameter  study,  we  used  the  best  (01247),  the  worst  (35689),  and  an  arbitrary  (25678) 
set  of  five  digits.  Results  are  listed  in  Table  IV.  Again  KAB  and  GRP  were  difficult  speakers, 
but  now,  many  other  speakers  contributed  to  the  errors.  Based  on  the  V FA  *FR  values  for  these 
tests,  the  verification  accuracies  obtained  by  representing  each  speaker  by  five  words  were 
significantly  worse  than  those  obtained  using  ail  10  words.  Generally,  the  degradation  in  perfor¬ 
mance  was  restricted  to  false  acceptances,  and  the  overall  performance  was  closer  to  the  design 
goal  of  equal  error  rates  for  the  two  types  of  errors. 

E.  Multisection  Results 

Parameter  Studies.  We  varied  both  the  section  length  n  and  the  section  codebook  rate  R$  in 
these  experiments.  Table  V  shows  the  results.  Generally  for  a  fixed  R$  value,  better  results  are 
achieved  using  smaller  n  values.  For  R$=  2,  tests  using  n  equal  to  1  and  2  were  not  done 
because  of  insufficient  codebook  training  data.  Using  n  =4  and  Rs  —  2,  we  tested  the  verification 
performance  of  the  individual  digits;  the  results  are  in  Table  VI.  Again  as  in  the  single  section 
approach,  no  single  digit  gives  good  overall  results  and  the  errors  are  biased  toward  false  rejec¬ 
tions. 

Verification  Tests.  We  used  n  =4  and  Rs  =2  for  the  verification  tests.  These  conditions  were 
chosen  because  they  did  well  both  in  the  parameter  study  and  in  previous  isolated  word  recogni¬ 
tion  work  [16].  Table  VII  contains  the  results  using  all  10  digits  to  make  the  verification  decision. 
No  speaker  was  particularly  difficult,  as  KAB  and  GRD  were  when  using  the  single  section 
approach,  and  in  general,  the  results  are  closer  to  the  design  goal  of  equal  false-rejection  and 
false-acceptance  error  rates  than  were  the  single  section  results.  VFA*FR  was  0.6. 

Again,  we  did  verification  tests  using  the  best  (12467),  the  worst  (03589),  and  an  arbitrary 
(01234)  set  of  five  digits;  the  results  are  in  Table  VIII.  The  verification  performance  of  the  vari¬ 
ous  five-digit  subsets  corresponded  well  with  the  expected  performance  based  on  the  single  digit 
study  -  i.e.,  the  best  five-digit  set  had  the  smallest  VFA *FR  ,  the  worst  set  had  the  largest 
VFA*FR  ,  and  the  arbitrary  set  had  an  V  FA  *FR  between  the  other  two.  The  only  consistently 
difficult  speaker  in  these  tests  was  KAB;  averaged  over  the  3  five-digit  tests,  he  had  a  false  accep¬ 
tance  rate  of  3.3%. 

F.  Matrix  Quantization 

Parameter  Study.  We  varied  the  codebook  rate  Ry  and  the  matrix  size  K  in  these  experi¬ 
ments.  For  each  K  value,  the  maximum  Ry  was  limited  by  the  amount  of  codebook  training 
data  (poor  codebooks  often  result  if  insufficient  training  data  is  used).  The  results  are  listed  in 
Table  DC.  No  obvious  relationship  between  K  and  R m  is  shown  in  these  results.  Using  Ry  =  3 
and  K  —  8  (these  conditions  are  also  good  for  isolated  word  recognition[l7]),  we  measured  the 
verification  performance  of  the  individual  digits;  these  results  are  in  Table  X.  As  in  the  single 
section  and  multisection  apf  roaches,  the  error  rates  are  biased  toward  false  rejections. 

Verification  Tests.  The  full  data  base  results  using  R ^  =  3,  K  =8,  and  all  10  digits  are  listed 
in  Table  XI.  Once  again,  KAB  was  a  difficult  speaker.  We  tested  the  best  (12467),  the  worst 
(03589),  and  an  arbitrary  (01234)  five  digits;  the  verification  results  are  in  Table  XII.  The  relative 
performance  of  the  five-digit  sets  did  not  correspond  exactly  with  the  expected  results  based  on 
the  individual  digit  performances,  but  the  worst  digit-set  did  produce  the  poorest  results. 
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V.  SUMMARY  AND  DISCUSSION 


The  verification  performance  (\/FA*FR  )  for  the  three  source  models  when  using  only  a  sin¬ 
gle  digit  per  speaker  were  similar  -  roughly  varying  from  4  to  8  depending  on  the  digit,  and  con¬ 
sistently,  the  digits  1,2,4  and  7  individually  did  best  in  the  speaker  verification  tests.  When  the 
individual  digits  were  joined  with  the  majority  rule  classifier,  however,  the  verification  perfor¬ 
mances  of  the  three  approaches  were  no  longer  equivalent.  The  multisection  VQ  source  model  did 
best  when  using  the  10-  and  5-digit  sets  of  verification  words  (for  the  10  digits,  s/'FA  *FR  =  0.6; 
for  the  best  5  digit  set,  \/ FA  *FR  —  0.7).  In  addition,  the  multisection  VQ  approach  came  closer 
to  satisfying  the  design  goal  of  equal  error  rates,  and  the  results  on  the  5-digit  subsets 
corresponded  more  closely  to  the  expected  results  (based  on  \/ FA  *FR  for  the  individual  digits). 
The  next  best  source  model  was  the  single  section  approach,  although  the  differences  in  V FA  *FR 
values  between  the  single  section  VQ  and  the  MQ  approach  were  small. 

All  three  source  models  did  well  in  the  speaker  verification  tests,  and  in  retrospect,  the  simi¬ 
larity  of  their  performances  is  not  surprising.  The  three  approaches  are  intimately  connected 
through  the  codebook  design  algorithm,  and  both  the  multisection  VQ  and  the  MQ  approaches 
are  generalizations  of  single  section  VQ.  This  can  be  seen  by  considering  the  multisection  VQ 
approach  with  a  section  length  n  equal  to  the  normalization  length  (24  in  this  study),  and  consid¬ 
ering  the  MQ  approach  with  the  matrix  size  K  equal  to  1.  Each  approach  reduces  to  single  sec¬ 
tion  VQ  under  the  appropriate  condition. 

The  single  section  VQ  source  model  captures  only  the  short-time  spectrum  shape  informa¬ 
tion.  This  spectrum  shape  information  is  useful  in  speaker  verification  because  it  contains  esti¬ 
mates  of  formant  frequencies,  relative  formant  amplitudes,  and  formant  bandwidths,  and  these 
are  correlated  with  the  locations  and  physical  sizes  of  the  speech  articulators.  As  such,  the  single 
section  results  are  a  measure  of  how  well  the  short-time  spectrum  can  characterize  a  speaker.  In 
addition,  because  the  codebook  spectra  are  unordered,  the  single  section  VQ  source  model  is 
directly  applicable  to  text-independent  speaker  verification  [18] .  It  is  generally  believed,  however, 
that  examining  parameters  as  a  function  of  time  is  valuable  in  speaker  verification  for  two  rea¬ 
sons;  (1)  many  of  the  speaker-characteristic  properties  of  speech  are  the  result  of  idiosyncrasies  in 
the  speaking  habits  of  people  and  (2)  by  considering  the  time  sequence  of  parameters,  the 
emphasis  is  on  how  the  parameters  vary  rather  than  the  exact  value  of  a  parameter.  The  mul¬ 
tisection  VQ  and  the  MQ  approaches  represent  two  different  ways  of  incorporating  some  phonetic 
duration  information  into  the  verification  process  while  maintaining  the  information-theoretic 
source  model  approach.  Multisection  VQ  improves  the  verification  performance,  and  at  best,  MQ 
does  not  degrade  the  performance.  It  is  unclear  why  the  durational  information  provided  by  the 
MQ  approach  does  not  improve  the  verification  performance. 

In  addition  to  phonetic  durations,  a  speaker  will  say  an  utterance  with  characteristic  tones 
or  intonations,  and  stresses  [28,  l] .  Because  these  are  roughly  independent  of  the  spectrum  shape 
information,  improvements  in  the  verification  accuracy  could  probably  be  achieved  by  adding 
pitch  and  short  time  energy  information  to  the  verification  process. 

As  an  aside,  it  is  interesting  to  consider  how  VQ  speech  coding  could  defeat  a  speaker 
verification  or  identification  system.  Our  single  section  VQ  results  show  that  the  source  model 
found  by  using  the  Linde,  Buzo,  and  Gray  clustering  algorithm  [12]  is  an  accurate  representation 
of  the  short-time  spectra  produced  by  a  speaker.  To  impersonate  a  speaker,  one  needs  only  to 
obtain  training  data  spoken  by  that  speaker  and  to  design  a  VQ  codebook  specifically  for  that 
speaker.  Anyone  could  talk  through  this  codebook  (via  VQ  speech  coding),  and  the  resulting 
speech  would  be  characteristic  of  the  speaker  who  provided  the  training  data.  It  seems  that  this 
procedure  would  defeat  any  speaker  recognition  system  that  relies  solely  on  short-time  spectrum 
representations. 

Finally,  the  connection  between  these  speaker  verification  approaches  and  our  previous  iso¬ 
lated  word  recognition  approaches  needs  to  be  emphasized.  The  parameters  (codebook  size,  sec¬ 
tion  length,  and  matrix  size)  and  the  source  model  (codebook)  design  procedure  used  in  each  of 
the  speaker  verification  tests  are  exactly  those  used  in  our  previous  work  on  isolated  word 


recognition  [9,  16,  17].  In  those  studies,  accuracies  for  speaker  trained  recognition  of  the  digits 
exceeded  99%.  The  very  good  speaker  verification  and  isolated  word  recognition  results  achieved 
using  these  approaches  point  toward  a  combined  speaker-speech  recognition  system.  These 
results  also  illustrate  the  power  of  the  VQ  source  coding  approach  using  the  Linde,  Buzo,  and 
Gray  clustering  algorithm  [12]. 
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#  OF  TRAINING  UTTERANCES 

Figure  1.  The  codebook  design  distortion  for  each  of  the  10  digits  as  a  function  of  the  number  of 
codebook  training  utterances  -  one  speaker’s  results. 
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Table  I.  Speaker  Verification  Study:  Single  Section  Codebooks, 
Majority  Rule,  And  All  10  Digits. 


Codebook 

Number  Of 
^moosteMJtterances 

False 

^Acceotancw^ 

Number  Of 
Admissible  Utterances 

False 

Rejections 

1 

800 

m 

l 

112 

15 

2 

800 

1 

112 

7 

3 

800 

0 

112 

ae 

4 

4 

800 

0 

112 
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Table  IL  Speaker  Verification  Study:  A  Single  Digit 
Codebook  Per  Speaker  And  R  =*»  3. 


Digit 

Spoken 

Number  Of 
Imposter  Utterances 

HKPSfll 

Number  Of 
Admissible  Utterances 

False 

Rejections 

V'fA  *FR 

ZERO 

800 

24  (3.035) 

112 

10  (8.9%) 

5.2 

ONE 

800 

9  (1.135) 

112 

19  (17.035) 

4.3 

TWO 

800 

14  (1.835) 

112 

10  (8.935) 

4.0 

800 

17  (2.135) 

112 

17  (15.235) 

5.7 

FOUR 

800 

12  (1.535) 

112 

13  (11.635) 

4.2 

FIVE 

800 

19  (2.435) 

112 

26  (23.235) 

7.5 

SIX 

800 

14(1.835) 

112 

17  (15.235) 

5.2 

SEVEN 

800 

9(1.135) 

112 

16  (14.335) 

4.0 

EIGHT 

800 

32  (4.035) 

112 

15  (13.435) 

7.3 

800 

20  (2.535) 

112 

20  (17.935) 

6.7 

4.0 

7.3 

6.7 
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Table  m.  Speaker  Verification  Results:  R  =  3,  Majority  Rule,  and  All  10  Digits. 


Speaker 

ID 

Number  Of 
Imposter  Utterances 

False 

Acceptances 

Number  Of 
Admissible  Utterances 

False 

Rejections 

11 

'JFA-FR 

TBS 

ooo 

0 

10 

0 

0.0 

WMF 

222 

0 

10 

0 

0.0 

RLD 

222 

1 

10 

0 

0.0 

GRD 

ooo 

1 

10 

1 

2.1 

KAB 

ooo 

5 

10 

o 

6.7 

MSW 

222 

0 

10 

0 

0.0 

REH 

ano 

0 

10 

0 

0.0 

RGL 

ooo 

0 

10 

0 

0.0 

CJP 

ooo 

0 

10 

0 

0.0 

DFG 

ooo 

0 

10 

0 

0.0 

ALK 

ooo 

0 

10 

0 

0.0 

HNJ 

2  22 

0 

10 

0 

0.0 

GNL 

222 

0 

10 

1 

0.0 

JWS 

222 

0 

10 

0 

0.0 

SJN 

222 

0 

10 

1 

0.0 

SAS 

222 

0 

10 

1 

0.0 

Totals 

3552 

7  (0.2%) 

160 

6  (3.3%) 

0.S) 

F£i 


Table  IV.  Speaker  Verification  Results:  Rate-3  Single  Section  Codebooks, 

Majority  Rule,  And  5  Digits. 


Digit 

Subset 

Number  Of 
Imposter  Utterances 

False  "1 

Acceptances 

Number  Of 
Admissible  Utterances 

False 

Rejections 

v  FA  * FR 

01247 

3552 

19(0.5%) 

160 

6  (3.8%) 

1.4 

35689 

3552 

27  (0.8%) 

160 

4  (2.5%) 

1.4 

25678 

3552 

25  (0.7%) 

160 

8  (5.0%) 

1.9 

Table  V.  Speaker  Verification  Study:  Multisection  Codebooks,  Majority  Rule, 
All  10  Digits,  800  Imposter  Utterances,  and  112  Admissible  Utterances. 


Codebook 
Rate  (/?? ) 

n  =12 

# FA  #  FR 

n 

#  FA 

=8 
#  FR 

n 

#  FA 

=4 
#  FR 

n 

*  fa 

— o 

*  FR 

n 

#  FA 

#  FR 

0 

18  19 

6 

9 

0 

10 

0 

10 

0 

10 

l 

1  6 

1 

6 

0 

8 

0 

5 

0 

5 

2 

0  7 

0 

6 

0 

5 

- 

- 

- 

- 

13 
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Table  VI.  Speaker  Verification  Study:  A  Multisection  Digit 
Codebook  Per  Speaker,  Rs  =  2,  and  n  =*  4. 


EBffli 

Number  Of 
Imposter  Utterances 

False 

Acceptances 

Number  Of 
Admissible  Utterances 

False 

Rejections 

v  FA mFR 

ZERO 

300 

i9  (2 .495) 

112 

12  (10.795) 

5.1 

ONE 

300 

11  (1.495) 

112 

13  (11.695) 

4.0 

TWO 

300 

8  (1.095) 

112 

16  (14.395) 

3.3 

THREE 

300 

21  (2.895) 

112 

18  (16.195) 

6.5 

FOUR 

300 

13  (1.895) 

112 

13  (11.695) 

4.3 

FIVE 

800 

18  (2.095) 

112 

31  (27.795) 

7.4 

SIX 

300 

9  (1.195) 

112 

IS  (13.495) 

3.3 

SEVEN 

300 

8  (0.895) 

112 

24  (21.495) 

4.1 

EIGHT 

300 

33  (4.195) 

112 

9  (8.095) 

5.7 

NINE 

800 

16  (2.095) 

112 

19  (17.095) 

5.8 

Table  VTI.  Speaker  Verification  Results:  Rs  =  2,  n 
Majority  Rule,  and  All  10  Digits. 


4, 


Speaker 

ID 

Number  Of 
Imposter  Utterances 

False 

Acceptances 

Number  Of 
Admissible  Utterances 

False 

Rejections 

.  FA  ’F 

TBS 

222 

l 

10 

0 

0.0 

WMF 

ooo 

0 

10 

0 

0.0 

RLD 

ooo 

0 

10 

0 

0.0 

GRD 

ooo 

l 

10 

0 

0.0 

ooo 

o 

10 

0 

0.0 

MSW 

OOO 

0 

10 

0 

REH 

OOO 

0 

10 

0 

0.0 

RGL 

OOO 

0 

10 

0 

0.0 

crp 

ooo 

o 

10 

0 

0.0 

DFG 

ooo 

0 

10 

0 

0.0 

AJLK 

ooo 

3 

10 

0 

0.0 

HNJ 

OOO 

0 

10 

0 

0.0 

GNL 

ooo 

0 

10 

o 

0.0 

JWS 

ooo 

0 

10 

0 

0.0 

SJN 

ooo 

0 

10 

0 

0.0 

SAS 

ooo 

0 

10 

0 

0.0 

Totals 

3552 

9  (0.395) 

160 

2(1.395) 

0.6 

14 


Table  VUI.  Speaker  Verification  Results:  Rate-2  Multisection  Codebooks 

Majority  Rule,  And  5  Digits. 


Digit 

Subset 


2467 

3589 

1234 


Number  Of 
Imposter  Utterances 


3552 

3552 

3552 


False 

Acceptances 


BUS 

.7%) 

SB 

.0%) 

17 

£ 

.5%) 

Number  Of 
Admissible  Utterances 


Table  IX.  Speaker  Verification  Study:  Matrix  Quantization  Codebooks,  Majority 
Rule,  All  10  Digits,  800  Imposter  Utterances,  and  112  Admissible  Utterances 


4 

#  fa  #fr 

K^S 

#  FA  #  FR 

/f=12 

#  FA  #  FR 

6  11 

0  10 

0  8 

5  16 

O'  8 

4  14 

0  9 

K  =24 

#  FA  #  FR 


2  8 


Table  X.  Speaker  Verification  Study:  A  Matrix  Quantization 
Digit  Codebook  Per  Speaker  And  RM  =3. 


Digit  Number  Of  False 

Spoken  Imposter  Utterances  Acceptances 
ZERO  300  24  (3.0%) 

ONE  300  0  (0.8%) 

TWO  800  12  (1.5%) 

THREE  800  29  (3.8%) 

FOUR  300  12  (1.5%) 

FIVE  800  45(5.6%) 

SIX  800  11  (1.4%) 

SEVEN  800  11  (1.4%) 

EIGHT  800  38  (4.5%) 

NINE _ 800 _ 17  (2.1%) 


Number  Of 
Admissible  Utterances 

False 

Rejections 

v FA  * FR 

112 

12  (10.7%) 

5.7 

112 

14  (12.5%) 

3.2 

112 

16  (14.3%) 

4.6 

112 

21  (18.3%) 

8.2 

112 

13  (11.6%) 

4.2 

112 

33  (29.5%) 

12.9 

112 

16  (14.3%) 

4.5 

112 

22  (19.6%) 

5.2 

112 

14  (12.5%) 

7.5 

112 

22  (19.6%') 

6.4 

15 


.  t; 


9 


|V.‘. 

■V-. 


Table  XI.  Speaker  Verification  Results:  RM  =  3, 
K  =  8,  Majority  Rule,  and  All  10  Digits. 


Speaker 

ID 

Number  Of 
Imposter  Utterances 

Number  Of 
Admissible  Utterances 

False 

Rejections 

y/ FA  *FR 

TBS 

ooo 

i 

10 

0 

0.0 

ooo 

0 

10 

0 

0.0 

RLD 

•lOO 

0 

10 

1 

o.o. 

GRD 

ooo 

1 

10 

1 

2.1 

KAB 

ooo 

3 

10 

3 

6.4 

MSW 

ooo 

0 

10 

0 

0.0 

REH 

ooo 

0 

10 

0 

0.0 

RGL 

ooo 

o 

10 

0 

0.0 

CJP 

ooo 

0 

10 

0 

0.0 

DFG 

ooo 

0 

10 

0 

0.0  | 

ALK 

222 

10 

0 

0.0 

222 

0 

10 

0 

0.0 

GNL 

222 

0 

10 

o 

0.0 

JWS 

ooo 

0 

10 

0 

0.0 

SJN 

ooo 

0 

10 

1 

0.0 

sas 

ooo 

0 

10 

l 

0.0 

Totals 

3552 

;  io.2%) 

1(50 

0  15.  ti'-p) 

_ n  1 
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